I need to import large (> 2Gb) delimited text files from federal databases. These are only available for download as delimited text files. I watched your Members Only example from your Import Data video on YouTube but it's intended for fixed-length field text files.
How do I accomplish the same thing using VBA to import my large DELIMITED data files into an Access table?
You'd have to use the same kind of file I/O but process each line as delimited data. I talk about File I/O in a lot more detail in Access Developer 30. You could also break that big file into smaller pieces and then do a regular import of that. Lots of ways to attack the problem.
Kevin Yip
@Reply 4 years ago
Assuming your text file has a line break separating each row, you can use the Line Input statement to read each line, E.g.:
Open "C:\myfile.txt" For Input As #1
Do While Not EOF(1)
Line Input #1, s
Loop
The Split() function can put delimited data into an array. E.g.:
Dim arr as Variant
arr = Split("Doe~John~56~9/11/1970~100 Main St", "~")
After running this, arr(0) equals to "Doe", arr(1) to "John", and so on, all strings.
Scott Axton
@Reply 4 years ago
No one has addressed the pink elephant in the room yet so here goes...
If your db is already at 1.5 gb in size you are not going to be able to import > 2 gb in data. You'll only be able to import .5 gb. Actually less I think because while the limit is 2 gb, I don't think you can use 2 gb, due to Access overhead.
Ok so let's create a brand new empty db, You still have some overhead from Access AND you have to tie it all together.
Now for pink elephant's little brother, blue elephant.
A delimited file is not going to have the wasted space like the example of the video where the fixed length file has wasted space of the "spaces".
Depending on what fields you actually keep / import you are going to get huge fast.
The likelihood of needing to up-size your db to SQL Server, in my mind, is looming fast if you are using data sets that large.
Good point, Scott. SOME of the fields MAY take less space. For example, a Long Int only takes 4 bytes. If you're storing a number like 3,423,214,231 in a text field, that's 10 bytes right there. So... you may get SOME compression. But yeah, you may definitely need to consider a bigger database - or splitting that file up into multiple tables (i.e. back-end ACCDB files).
Sorry, only students may add comments.
Click here for more
information on how you can set up an account.
If you are a Visitor, go ahead and post your reply as a
new comment, and we'll move it here for you
once it's approved. Be sure to use the same name and email address.
This thread is now CLOSED. If you wish to comment, start a NEW discussion in
Access Expert 20.