How to Combine CSV Files using Command

Combining CSV files is a common task when working with data in Linux. Whether you’re merging logs, consolidating reports, or preparing data for analysis, knowing how to combine CSV files quickly is a must-have skill. In this post, I’ll show you three easy ways to combine CSV files using Linux command-line tools: cat, awk, and paste. Let’s get started!

Why Combine CSV Files?

Combining CSV files is useful for:

• Merging data from multiple sources into a single file.
• Consolidating reports or logs for analysis.
• Preparing data for use in scripts or applications.

Method 1: Using cat (Simple Concatenation)

If your CSV files have the same structure (same columns in the same order), you can use cat to concatenate them:

cat file1.csv file2.csv > combined.csv

Explanation:

• cat: Concatenates files.
• file1.csv file2.csv: The files to combine.
• > combined.csv: Saves the output to a new file.

Example:

If file1.csv contains:

Name,Age,Location
John,30,New York

And file2.csv contains:

Name,Age,Location
Jane,25,Los Angeles

The output (combined.csv) will be:

Name,Age,Location
John,30,New York
Name,Age,Location
Jane,25,Los Angeles

Method 2: Using awk (Remove Duplicate Headers)

If your CSV files have headers and you want to remove duplicate headers, use awk:

awk 'NR==1 || FNR!=1' file1.csv file2.csv > combined.csv

Explanation:

• NR==1: Prints the header from the first file.
• FNR!=1: Skips the header for all subsequent files.
• file1.csv file2.csv: The files to combine.
• > combined.csv: Saves the output to a new file.

Example:

Using the same file1.csv and file2.csv as above, the output (combined.csv) will be:

Name,Age,Location
John,30,New York
Jane,25,Los Angeles

Method 3: Using paste (Side-by-Side Combination)

If you want to combine CSV files side by side (e.g., merging columns), use paste:

paste -d',' file1.csv file2.csv > combined.csv

Explanation:

• paste: Combines files side by side.
• -d',': Sets the delimiter to a comma.
• file1.csv file2.csv: The files to combine.
• > combined.csv: Saves the output to a new file.

Example:

If file1.csv contains:

Name,Age
John,30

And file2.csv contains:

Location
New York

The output (combined.csv) will be:

Name,Age,Location
John,30,New York

Which Method Should You Use?

• cat: Use for simple concatenation of files with the same structure.
• awk: Use to remove duplicate headers.
• paste: Use for side-by-side combination of files.

Conclusion

Combining CSV files in Linux is easy with tools like cat, awk, and paste. Whether you’re merging rows or columns, these commands will save you time and effort. Try them out and see which one works best for your needs!

Got questions or need further clarification? Drop a comment below! And don’t forget to share this post if you found it helpful. 🐧✨