How to obtain high I/O performance in Windows for very large files

From Yggenyk
Jump to: navigation, search

I/O performance i Windows

Standard Windows 2003 I/O function calls for writing to disk are effective for small and moderate sized files, but deteriorate in performance as file sizes grow beyond 100 Gigabytes in size. Disk writes of large files writing directly to disk arrays using the Win32 API offers performance improvement of 10X over standard Windows function calls for real-time applications that require consistent performance over the entire file.

Best way to read binary files using .NET

If you need good performance for very large files you need to use using the Win32 API. It offers performance improvement of 10X if the files are huge.

Win32 file sample

Win32 File Samples article tries to explain how to read from and write to Disk files using Win32 functions. Though Win32 is not user-friendly and programmers are moving towards .Net, this could be useful in some places where a quick fix for old programs are needed.


The basic idea behind having the set of functions for manipulating the files in win32 was great. The same API was designed to be used for Named Pipes, Mail slots, Memory mapped files etc. The parameters are the only difference. The parameters will depend on the type of object to be opened.

This win32 file article explains how to make simple use the functions createfile, writefile and readfile for writing and reading disk files.

// Writing to a Win32 File: HANDLE hFile;
DWORD wmWritten;
char strData[] = "Test data written to explain the win32 file sample";
hFile = CreateFile("C:\\test.txt",GENERIC_READ|GENERIC_WRITE,
FILE_SHARE_READ,NULL,OPEN_ALWAYS,FILE_ATTRIBUTE_NORMAL,NULL);
WriteFile(hFile,strData,(DWORD)(sizeof(strData)),&wmWritten,NULL);
CloseHandle(hFile);

// Reading data from a Win32 File:

// This needs the file to be opened in read mode (to allow the program a read access) and
// then call the win32 file function ReadFile with the address of the character array/pointer
// where the data is to be read. The sample code for the same is given as below.

HANDLE hFile;
DWORD wmWritten;
char strVal[1024];
hFile = CreateFile("C:\\test.txt",GENERIC_READ|GENERIC_WRITE,
FILE_SHARE_READ,NULL,OPEN_ALWAYS,FILE_ATTRIBUTE_NORMAL,NULL);
ReadFile(hFile,strVal,1024,&wmWritten,NULL);
CloseHandle(hFile);

It is better to forget using these functions and move towards using MFC classes CFile or CStdioFile for file handling.

This needs the file to be opened in Write mode (to allow the program a write access) and then call the win32 file function WriteFile with the data to be written. The sample code for the same is given as below.

id=siteTree