If you've got here from Google, it's probably because you've exported a CSV or TXT file from picoscope and it's come out with 1048579 lines - maybe 3 of which are headers and 1048576 are data. You can find several posts on their forum about the same thing.

Seems like Excel worksheets are limited to 1,048,576 rows by 16,384 columns, and Picoscope silently truncates CSVs to that length. I'm not sure why they decided to truncate silently, rather than giving the user the option to decimate the data or choose a window to export, or putting a note on the last line of the data saying that it had been truncated and why.

Picoscope support recommends exporting in Matlab format instead. But what if you really want a big CSV, and you don't have Matlab because it costs £1,600 a seat which is a lot of money for a single CSV? Python to the rescue!

Converting picoscope output from .mat to .csv

Here's the Python script I used to do the conversion. My install of Ubuntu 16.04 already had Python 2.7.12 installed, but I had to run sudo apt-get install python-scipy

import scipy.io
import numpy
mat = scipy.io.loadmat('20160918-0002.mat')
startTime = mat['Tstart'][0][0]
numSamples = mat['Length'][0][0]
endTime = startTime+numSamples*mat['Tinterval'][0][0]
times = numpy.linspace(startTime, endTime, numSamples).reshape(numSamples,1)
arrayForCsv = numpy.concatenate((times, mat['A'], mat['B'], mat['C'], mat['D'], mat['E']), axis=1)
numpy.savetxt('20160918-0002.csv.gz', arrayForCsv, delimiter=',', fmt='%.9f')

Note that you'll have to change the loadmat and savetxt lines with the filenames you want, and change the concatenate line if you weren't using five channels. Simply change the mat['A'], mat['B']...mat['E'] section to list the channels you want in your CSV.



Published

18 September 2016

Tags

website@mjt.me.uk · Home · Archive · Tags