IPhone, IPad, IPod: HTTP Streaming with segmenter and m3u8 on Windows

So, 23.05.2010 - 17:07 -- Daniel Espendiller

With the new Ipod touch, iPad, IPhone OS3 Apple implemented HTTP Streaming. A very tricky way but works very good without any flash usage. A file or a stream will split to several smaller parts that will put together in a playlist. On live streaming the playlist will updated automatically. To get this, videos must trancoded and chuked. On google you will find a open source file segmenter.c and a apple tool mediastreamsegmenter for Mac OS only. So we must use segmenter.c and compile it. Since most users will do that on Linux we will not find any Windows files. So i have compiled a exe file for my home streaming.

Alternative: Air Video

You can also use the Air Video for streaming your files. There you get a server software for Windows and a patched ffmpeg. Some guys made a mod of this and patched that patched ffmpeg again :) so you can use streaming links (i.e from dbox, dreambox) in this software. Because this was too complex i decided to do something on my known.

Download

In the Sidebar you can find the Download archive link of the binary files for Windows. I host them now on Google Code (httpsegmenter). I tested in on Windows XP and Windows 7 and in works after patching arround in the source file (segmenter.c) (see above). The commandline syntax is similar to the orginal one but has an additional optional paramter at the end. But you can use all scripts and commandlines snippets without changing anything.
So my building system works very well now, i also put a Windows binary of live_segmenter into the download archive. You can upload the stream to Amazon S3, ftp, shh , .... Not tested yet.

In the download archive you will find a batch file for starting a stream. Some syntax as on Linux System, so Google for more information or look here if you need help.

ffmpeg -i C:\your\nice\video\source.avi -f mpegts -acodec libmp3lame -ab 64000 -s 480x320 -vcodec libx264 -b 480000 -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16  -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 400k -maxrate 524288 -bufsize 524288 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 480:320 -g 30 -async 2 - | segmenter - 10 stream/stream stream/stream.m3u8 http://localhost/ 5 1

More Information

  • Think of using Apache or any other Webserver to get the playlist open on your Ipod touch, iPad or IPhone. VLC should also open then m3u8 file but i have not tested yet.
  • In the archive you can see a ffmpeg binary. Some ffmpeg Version doesnt implement the necessary codec, this does

Usage: c:\...\segmenter <input MPEG-TS file> <segment duration in seconds> <output MPEG-TS file prefix> <output m3u8 index file> <http prefix> [<segment window size>] [<search kill file>]
 
Compiled by Daniel Espendiller - www.espend.de
build on May 23 2010 16:55:37 with 3.4.5 (mingw-vista special r3)
 
Took some code from:
 - source:http://svn.assembla.com/svn/legend/segmenter/
 - iStreamdev:http://projects.vdr-developer.org/git/?p=istreamdev.git;a=tree;f=segmenter;hb=HEAD
 - live_segmenter:http://github.com/carsonmcdonald/HTTP-Live-Video-Stream-Segmenter-and-Distributor

For Experts only

Patched segmenter

To work in Windows i hade to patch the function write_index_file because the m3u8 playlist generator doesnt remove older files on a Windows System (does rename only). Furthermore i added several tweaks i a got from comments and user sites to clean up the code. Now a PID file will generated and you can close a running streaming progress by putting a file with name kill in the same folder as the binary.

int write_index_file(const char [...]
  [...]
  remove(index);
  if (rename(tmp_index, index)) {
    fprintf(stderr, "Could not rename %s to %s\n", tmp_index, index);
    return -1;
  }
 
  return 0;
}

Compile

First of all you need the mingw32 and mssys buliding system, simply use the setup files to install them on windows. Moreover you need the ffmpeg libraries. Its very tricky to get them compiled but you can find alot on the FFmpeg on Windows. If you installed the building system you must replace serveral binaries and implement new versions i.e. bash-3.1.17-2-msys-1.0.11, coreutils-5.97-MSYS-1.0.11, make-3.81-2-msys-1.0.11-bin some informationen you can find here: gooli.org - Building ffmpeg for Windows with MSYS and MinGW. I cannot tell that in detail here.
You dont need to build the full ffmpeg with libx264 and other codec, when you take a different ffmpeg binary (mostly static) use only configure --enable-memalign-hack to start your ffmpeg builing and then do a make.

After hopefully you get a working ffmpeg build process you can build segmenter.c with gcc. If you check out the orignal source you can find a makefile in that. Some people tweaks alot on that but we dont need this. To compile the file simply use this:

gcc -Wall -g segmenter.c -o segmenter.exe -L /usr/local/lib -lavformat -lavcodec -lavutil

It should work hopefully without any warning. Under mingw32 you can now run the exe binary. To use it in directly in windows put the ffmpeg dll files in the some folder as binary it should be: avcodec-xx.dll, avformat-xx.dll, avutil-xx.dll find them in ffmpeg source folder after ffmpeg build. A static build without the dlls didnt work for me right now.

Ready build-enviroment for you

After some requests i but a full working and patched mingw32 / mysys enviroment as mentioned above into the the Google Project Download list

VideoLAN (VLC)

The newest Version of VLC can also stream to IPhone, IPad and others with http segmentation (some solution mentioned above). Look at the wiki: Documentation:Streaming HowTo/Streaming for the iPhone

Disqus - noscript

Nice job.

I'd like to know the correct command line when you want to segment a stream, not a file.

Thanks!
you can find an example in zip file. there is a file called start.bat which contains the command line. simply replace the filename with the stream url. think of using the newest ffmpeg version which can play more video format.

moreover look in the sm_stream_iphone module for drupal which i created to stream onlinevideo. you can find it here: http://svn.espend.de/filedetails.php?repname=st...

In the function sm_stream_iphone_stream() i use some commandlines which does exactly what you want.
Hi
I would like to get know , Which Open source segmenter was used for Iphone 2.0 to Chop vedios.
I am in hurry. Looking forward for your quick reply.

to the author :

a request
------------
It would be nice , if you can publish/post a snapshot(zip file) of your local build environment for building ffmpeg and segmenter along with their source code. I think a zip of your local c:\msys and c:\mingw c:\ffmpeg_src and any environment variables like PATH, registry keys and other common things needed to build these will be useful and helpful

The reason i am requesting this - I want to make minor changes to segmenter.c and build it for windows(segmenter.exe) but i am having trouble building ffmpeg and segmenter in windows. I tried these...

1) http://ffmpeg.arrozcru.org/wik... which is for cross-building ffmpeg for windows in ubuntu/linux.
But unfortunately the author has only a 64-bit solution and 32-bit support is just not there or not explained/tested properly

2) your suggested site - http://www.gooli.org/blog/buil...

this also is so outdated and does not work because mingw/msys has evolved largely to newer versions than explained here

thanks
alex
i uploaded the build environment to the project at http://code.google.com/p/https....

i didnt find any path variable. it should work without one.

README.txt
---------------------
as requested my build environment to build the segmenter on win32. (i used windows7 x86)

extract file to the root of C:\

includes MinGW and msys both patched we some extra packages
(e.g. bash-3.1.17-2-msys-1.0.11, coreutils-5.97-MSYS-1.0.11, make-3.81-2-msys-1.0.11-bin)
find more on http://www.espend.de/node/160

in ffmpeg-checkout-2010-05-22 you can find a full build ffmpeg version with the necessary libraries
put your source code to compile here

to build your code.
- run the batch file:
C:\msys\1.0\msys.bat

- change dir to:
cd /c/ffmpeg-checkout-2010-05-22

- compile file:
gcc -Wall -g segmenter.c -o segmenter.exe -L /usr/local/lib -lavformat -lavcodec -lavutil

- finish
you should have a exe file!



thanks a lot. segmenter builds like a charm. Yet to make changes and rebuild. Will provide more feedback then.

Never expected such a quick professional , prompt response from an open source developer! :-)

Puts to shame, paid developer support from big companies like msft, intel etc.,

--alex
I would like to know the way to segment vedios in Linux server. How to compile Segmenter.c with FFMPEG?
From where i can check that Zip file, I am not able to see it in this post.
From where i have to check Zip file? I am not able to see it on this post,
Hello there,

nice job! Great tool. Mediastreamsegmenter on the MAC is o.k. but MAC only which makes me favour your segmenter.
One question left. I wonder how you're doing the playlist generation in detail which means how can I get a playlist with correct durations in the end. The last .ts-segment tends to be smaller than the given value like 10 or 8 seconds. MACs Mediastreamsegmenter seems to catch the correct information and lists the last entry in the playlist with a shortened time value. See example here (generated with MACs Mediastreamsegmenter, official version from connect.apple.com from June 2010):
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10,
http://.../fileSequence0.ts
#EXTINF:10,
http://.../fileSequence1.ts
#EXTINF:10,
http://.../fileSequence2.ts
#EXTINF:3,
http://.../fileSequence3.ts
#EXT-X-ENDLIST

You see, the last fileSequence is given 3 seconds. The video itself is between 32 and 33s long.

Greetings,

Tobias
Thanks so much for providing this segmenter. It is working great! I'm using it to generate 4 playlists of varying quality. There are two very minor issues that I've seen:

1) The m3u8 file generated only provides absolute paths. Apple's medistreamvalidator complains about this. I worked around it with a simple sed script, but it would be nice to have an option to avoid outputting these paths... maybe just the lack of the URL could indicate that no URL prefix is necessary.

2) I'm seeing this warning, also returned from the mediastreamvalidator on a 20 minute video: "WARNING: Timestamps are too spread out across the variant playlists, with a maximum spread of 60.09 seconds and a maximum target duration of 10 seconds (threshold is twice that number)."

I suspect this timestamp warning has to do with the segments not being at exactly 10 seconds. I know the segmenter has to carefully pick where to segment... I've adjusted for keyframes every second on the second. Is there anything else I can be doing?
Hi, thanks for compiling the windows version. However, the example you provided in the start.bat file wasnt able to create a a valid stream.m3u8 file. It creates a .stream.m3u8 with 0 bytes thats gets timestamp updated, but nothing more.

.ts files is created pefektly.
same here. some new changes in the lastest revisions break that.
you can use:
segmenter-r6.7z or segmenter-r7-with-audio-only.7z

i will fix that bug later...
r14 should work proper now.
Thanks :)


Have you tried to transcode HD channels from Dreambox using this method ?
Transcoding SD channels is no problem at all, but HD channels doesnt seem to
work
Great introduction to HTTP streaming. I'm using VLC to transcode an wmv stream into mp4 and your precompiled segmenter (static r14), but I'm having problem since this is a live stream. The ipad/iphone stops after playing the files in the m3u8 file unless it does not contain #EXT-X-ENDLIST, in that case it understands that this is a live event and continuously reloads the file.

Can you please add an option to omit the #EXT-X-ENDLIST at the end of the file?
now use r15 :)

i removed the line "#EXT-X-ENDLIST" completely as it was in the lastest rev, too.

it helped me ...
hd should work well, but you need much cpu power. i tested it with "dvbviewer" that provide also a hd stream.
Why did you remove #EXT-X-ENDLIST? Now that it's not there our idevices don't know the length of the stream and don't show the progress bar properly anymore. We no longer have the ability to jump around in the file when it's not a live stream.

Right now we have to back batch the #EXT-X-ENDLIST to the end of the file after running it through the segmenter. You should add a command line option to turn it on and/or off instead of just removing it.
i am not sure why but the segmenter seems to not be putting out a correctly formatted playlist. i tried showing it here but it doesn't work here.

its not doing line breaks and its all bunched up and has a square in the file. which is a decoding errro are you using r\n\ ?? for windows it needs to be that way to break a line.

also what does the 5 and 1 stand for?

thanks for you help been stranded on this for days

after looking at the source code, yes that is the issue....

your using \n to break the lines and not r\n\ which is how Windows needs it to be done. is there anyway for this to be fixed?
Hi,

First off, Thank you so much for building this segmenter! 

However, I am having a problem, where I have videos that are a certain length, but when I run the segmenter, it isn't creating enough segments.  For example, I have a .ts file (created with ffmpeg) that is 558 seconds, but when I segment it with 10 second segments, I only get 41 segments!  I've seen this on every file i've tested today.  I am desperate for a fix.  Can you please reply to me ASAP at MRichards@OriginDigital.Com ?

This is a very interesting project. I am trying to segment mp3 files (2-3 hours in length) for http live streaming on demand. I have tried r19 static, but the ts files created don't validate. Apple's validator says:

2012-04-13-07-00-00-390.ts:WARNING: Media segment exceeds target duration of 10.00 seconds by 1024.91 seconds (segment duration is 1034.91 seconds)

these were created with:
segmenter -i filename.mp3 -d 10 -o prefixname -x playlist.m3u8

Is this segmenter able to handle mp3 files? It would be great if it could because I don't have easy access to a windows compiler, but for various reasons I have to do this task on windows right now.

Hello, there.

I create HLS content used by ffmpeg.exe and segmenter.exe.
Duration of the interval of the playlist that was generated but is not uniform, what would some possible causes?

Version of the tool that I have used is the following.
ffmpeg : N-47062-g26c531c (release Nov 25 2012)segmenter : 4.4.0 (release Feb 13 2012)

Hei, Nice job. But Im not get this do work. I downloaded segmenter r6 to windows xp. Then change the bat file to...

ffmpeg -i C:\Documents and Settings\stoperator\Desktop\MEELISE KAMA\segmenter-r6\Market 2.avi -f mpegts -acodec libmp3lame -ab 64000 -s 480x320 -vcodec libx264 -b 480000 -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 400k -maxrate 524288 -bufsize 524288 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 480:320 -g 30 -async 2 - | segmenter - 10 stream/stream stream/stream.m3u8 http://localhost/ 5 1

If i start .bat file or segmenter.exe then nothing happens. What i missed?

Regards

Meelis

Could I use "segmenter.exe" split audio file to m3u8 on windows for iPhone audio player????
Thanks!

Works nicely! Is there a static version of segmenter for audio-only TS? I found the stream version but not static. Thanks much

Segmenter R21 in the msys_mingw_segmenter_win32_buildroot.7z compiler can not, who can send me the R21 segmenter windows in the new environment, or the compiled file. Thank you
e-mail:273027946@qq.com