Page 1 of 2
Downloading large files
Posted: Sun Apr 27, 2014 2:57 pm
by Derrik
I've got some code which can create and send an HTTP request but I'm having trouble saving the response.
Here's a snippet of my code:
Code: Select all
char doneHeader = 0;
int recvd_len;
char incoming_buffer[BUFFER_SIZE];
int dataIn = 0;
while((recvd_len = recv(my_socket, incoming_buffer, BUFFER_SIZE, 0)) != 0) {
if(recvd_len > 0) {
int contentLength = recvd_len;
char *contentBuffer = incoming_buffer;
if(!doneHeader) {
contentLength -= getDataStart(incoming_buffer) - incoming_buffer;
contentBuffer = getDataStart(incoming_buffer);
doneHeader = 1;
}
FILE *o = fopen(localFilename, "ab");
fwrite(contentBuffer, contentLength, 1, o);
fclose(o);
dataIn += BUFFER_SIZE;
printf("%d / 1,424,880\n", dataIn);
}
}
iprintf("Done!\n");
What I am trying to do is download the file in chunks of BUFFER_SIZE (which is currently 1024) by appending the response to the end of the file each time new data is received.
The problem is, my DS outputs this:
1024 / 1,424,880
2048 / 1,424,880
...
105472 / 1,424,880
And then freezes.
My guess is that the DS is running out of RAM because dswifi isn't freeing the data that I've already written to the file and don't need anymore.
So how can I get dswifi to download the file in chunks, freeing up memory once it has been delt with?
Re: Downloading large files
Posted: Sun Apr 27, 2014 3:05 pm
by Derrik
OK; I ran the program again and got to "267264 / 1,424,880" before freezing.
What's going on?
Re: Downloading large files
Posted: Mon Apr 28, 2014 12:25 pm
by WinterMute
Opening and closing the file for each iteration of the loop probably isn't a good idea although it should work in theory.
There's no guarantee that recv will wait for BUFFER_SIZE bytes before returning, you should loop until the buffer is full if you want to save in BUFFER_SIZE chunks. You could also just use recvd_len as the number of bytes to write - libfat will buffer before writing to SD anyway.
Re: Downloading large files
Posted: Mon Apr 28, 2014 4:43 pm
by Derrik
Thanks, I've modified my code to only open and close the file once:
Code: Select all
FILE *o = fopen(localFilename, "ab");
while((recvd_len = recv(my_socket, incoming_buffer, BUFFER_SIZE, 0)) != 0) {
if(recvd_len > 0) {
int contentLength = recvd_len;
char *contentBuffer = incoming_buffer;
if(!doneHeader) {
contentLength -= getDataStart(incoming_buffer) - incoming_buffer;
contentBuffer = getDataStart(incoming_buffer);
doneHeader = 1;
}
fwrite(contentBuffer, contentLength, 1, o);
dataIn += BUFFER_SIZE;
printf("%d\n", dataIn);
}
}
iprintf("Done!\n");
fclose(o);
It usually is contentLength usually is recvd_len, apart from the first receive which needs to take away the size of the HTTP header.
I'm not really sure what else to try, it is still freezing at arbitrary points. The protocol is TCP so I shouldn't need to do my own timeout -> retry code should I?
Re: Downloading large files
Posted: Mon Apr 28, 2014 5:41 pm
by Derrik
My code is a lot more robust, I use the HTTP command "Range: bytes=start-end\r\n" in my header so I can download the file from the server in parts.
The code all works, it's just that it randomly freezes.
In the myConnect function I set my socket to be nonblocking:
Code: Select all
fcntl(my_socket, F_SETFL, O_NONBLOCK);
So the code shouldn't just be waiting for recvd function, since it is nonblocking. However once the application freezes I cannot get anything else to happen, it is frozen.
The only thing I can think of is that dswifi doesn't use this method to be nonblocking:
Code: Select all
fcntl(my_socket, F_SETFL, O_NONBLOCK);
And that my code hasn't frozen, it is just stuck at recv function because my sockets are blocking.
Please confirm that this code should make my sockets non blocking.
I tried this:
Code: Select all
u_long one = 1;
ioctlsocket(my_socket, FIONBIO, &one);
But I get compile errors (can't find ioctlsocket or FIONBIO).
Re: Downloading large files
Posted: Mon Apr 28, 2014 6:42 pm
by Derrik
Found it, it was:
ioctl()
Not ioctlsocket()
Like on Windows.
Re: Downloading large files
Posted: Tue Apr 29, 2014 4:36 pm
by Derrik
So now my sockets are non-blocking, I can see if it has been 5 seconds without receiving data, if so, retry.
Everytime I retry, it fails to download any more data.
It will go to like:
Downloading part 73...
Downloading part 74...
Downloading part 75...
Timout!
Downloading part 75...
Timout!
Downloading part 75...
...
Note, the amount of data changes each time, sometimes it makes it to part 63, other times, 75, other times 84, etc...
Is there some kind of way of flushing dswifi buffers, so that when I get to this point, I can totally retry from the beginning.
Re: Downloading large files
Posted: Tue Apr 29, 2014 5:29 pm
by Derrik
I've even tried this:
Code: Select all
shutdown(my_socket, 0);
closesocket(my_socket);
Wifi_DisconnectAP();
if(!Wifi_InitDefault(WFC_CONNECT)) {
printf(" Failed to connect to WiFi!\n");
}
printf(" Reconnected to WiFi\n");
myConnect(host);
printf(" Reconnected to host\n");
But it's not enough to get the DS to start receiving data again.
Re: Downloading large files
Posted: Tue Apr 29, 2014 5:30 pm
by Derrik
Although using the code from hbmenu to reboot the NDS file, and the DS can receive data again.
I've tried to take the code from hbmenu that resets cache so that the DS can receive data again, but it is hard to isolate the WiFi reset code.
Re: Downloading large files
Posted: Tue Apr 29, 2014 6:09 pm
by Derrik
OK; when I try to connect after recreating the socket (after shutting it down, closing it, calling socket) I get errno 119:
Code: Select all
errno = 0;
if(myConnect() == SOCKET_ERROR) {
printf(" connect() failed (%d)!\n", errno);
}
(myConnect just returns what connect() returns).
I've looked up the error code:
Code: Select all
#define ENAVAIL 119 /* No XENIX semaphores available */
What does this mean?