Problem / Solution
One issue with importing data into SQL Server is that the import file has rows that have too many or too few columns and therefore the imported data doesn’t insert correctly or doesn’t import at all. With small files this is easy to manually correct prior to import, but what about files with thousand or millions of rows? In this video tip, we look at how to use PowerShell to remove the bad records so the import process can function without issue.
Key Learning Items
- The import behavior of BULK INSERT when there are too few or too many delimiters in a line.
- PowerShell script to automate removing invalid lines.
- Reminder of why logging good data is important.
Read Companion Tip
Tim consults for FinTek Development and teaches the course Automating ETL on Udemy. He’s worked with technology since high school, helping his school win its first TCEA award and continues to work in automation, data architecture, back-end development, and smart contract architecture. Tim enjoys testing new technologies early in the diffusion of innovation curve and was an early adopter of NoSQL and smart contract development. He has a blog at http://www.fintekdev.com/ and helps contribute to local technical and financial events in Texas.
