/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */ /* vim:set ts=2 sw=2 sts=2 et cindent: */ /* ***** BEGIN LICENSE BLOCK ***** * Version: ML 1.1/GPL 2.0/LGPL 2.1 * * The contents of this file are subject to the Mozilla Public License Version * 1.1 (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * http://www.mozilla.org/MPL/ * * Software distributed under the License is distributed on an "AS IS" basis, * WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License * for the specific language governing rights and limitations under the * License. * * The Original Code is Mozilla code. * * The Initial Developer of the Original Code is the Mozilla Corporation. * Portions created by the Initial Developer are Copyright (C) 2007 * the Initial Developer. All Rights Reserved. * * Contributor(s): * Chris Double * * Alternatively, the contents of this file may be used under the terms of * either the GNU General Public License Version 2 or later (the "GPL"), or * the GNU Lesser General Public License Version 2.1 or later (the "LGPL"), * in which case the provisions of the GPL or the LGPL are applicable instead * of those above. If you wish to allow use of your version of this file only * under the terms of either the GPL or the LGPL, and not to allow others to * use your version of this file under the terms of the MPL, indicate your * decision by deleting the provisions above and replace them with the notice * and other provisions required by the GPL or the LGPL. If you do not delete * the provisions above, a recipient may use your version of this file under * the terms of any one of the MPL, the GPL or the LGPL. * * ***** END LICENSE BLOCK ***** */ /* Each video element has one thread. This thread, called the Decode thread, owns the resources for downloading and reading the video file. It goes through the file, decoding the theora and vorbis data. It uses Oggplay to do the decoding. It indirectly uses an nsMediaStream to do the file reading and seeking via Oggplay. All file reads and seeks must occur on this thread only. It handles the sending of the audio data to the sound device and the presentation of the video data at the correct frame rate. When the decode thread is created an event is dispatched to it. The event runs for the lifetime of the playback of the resource. The decode thread synchronises with the main thread via a single monitor held by the nsOggDecoder object. The event contains a Run method which consists of an infinite loop that checks the state that the state machine is in and processes operations on that state. The nsOggDecodeStateMachine class is the event that gets dispatched to the decode thread. It has the following states: DECODING_METADATA The Ogg headers are being loaded, and things like framerate, etc are being decoded. DECODING_FIRSTFRAME The first frame of audio/video data is being decoded. DECODING Video/Audio frames are being decoded. SEEKING A seek operation is in progress. BUFFERING Decoding is paused while data is buffered for smooth playback. COMPLETED The resource has completed decoding. SHUTDOWN The decoder object is about to be destroyed. The following result in state transitions. Shutdown() Clean up any resources the nsOggDecodeStateMachine owns. Decode() Start decoding video frames. Buffer This is not user initiated. It occurs when the available data in the stream drops below a certain point. Complete This is not user initiated. It occurs when the stream is completely decoded. Seek(float) Seek to the time position given in the resource. A state transition diagram: DECODING_METADATA | | Shutdown() v -->-------------------->--------------------------| | | DECODING_FIRSTFRAME v | | Shutdown() | v >-------------------->--------------------------| | |------------->----->------------------------| v DECODING | | | | | ^ v Seek(t) | | | | | Decode() | v | | | ^-----------<----SEEKING | v Complete v v | | | | | | | | | COMPLETED SHUTDOWN-<-| ^ ^ | |Shutdown() | | | | >-------->-----^ | Decode() |Seek(t) |Buffer() | -----------<--------<-------BUFFERING | | ^ v Shutdown() | | | ------------>-----| The Main thread controls the decode state machine by setting the value of a mPlayState variable and notifying on the monitor based on the high level player actions required (Seek, Pause, Play, etc). The player states are the states requested by the client through the DOM API. They represent the desired state of the player, while the decoder's state represents the actual state of the decoder. The high level state of the player is maintained via a PlayState value. It can have the following states: START The decoder has been initialized but has no resource loaded. PAUSED A request via the API has been received to pause playback. LOADING A request via the API has been received to load a resource. PLAYING A request via the API has been received to start playback. SEEKING A request via the API has been received to start seeking. COMPLETED Playback has completed. SHUTDOWN The decoder is about to be destroyed. State transition occurs when the Media Element calls the Play, Seek, etc methods on the nsOggDecoder object. When the transition occurs nsOggDecoder then calls the methods on the decoder state machine object to cause it to behave appropriate to the play state. The following represents the states that the player can be in, and the valid states the decode thread can be in at that time: player LOADING decoder DECODING_METADATA, DECODING_FIRSTFRAME player PLAYING decoder DECODING, BUFFERING, SEEKING, COMPLETED player PAUSED decoder DECODING, BUFFERING, SEEKING, COMPLETED player SEEKING decoder SEEKING player COMPLETED decoder SHUTDOWN player SHUTDOWN decoder SHUTDOWN The general sequence of events with these objects is: 1) The video element calls Load on nsMediaDecoder. This creates the decode thread and starts the channel for downloading the file. It instantiates and starts the Decode state machine. The high level LOADING state is entered, which results in the decode state machine to start decoding metadata. These are the headers that give the video size, framerate, etc. It returns immediately to the calling video element. 2) When the Ogg metadata has been loaded by the decode thread it will call a method on the video element object to inform it that this step is done, so it can do the things required by the video specification at this stage. The decoder then continues to decode the first frame of data. 3) When the first frame of Ogg data has been successfully decoded it calls a method on the video element object to inform it that this step has been done, once again so it can do the required things by the video specification at this stage. This results in the high level state changing to PLAYING or PAUSED depending on any user action that may have occurred. The decode thread, while in the DECODING state, plays audio and video, if the correct frame time comes around and the decoder play state is PLAYING. a/v synchronisation is done by a combination of liboggplay and the Decoder state machine. liboggplay ensures that a decoded frame of data has both the audio samples and the YUV data for that period of time. When a frame is decoded by the decode state machine it converts the YUV encoded video to RGB and copies the sound data to an internal FrameData object. This is stored in a queue of available decoded frames. Included in the FrameData object is the time that that frame should be displayed. The display state machine keeps track of the time since the last frame it played. After decoding a frame it checks if it is time to display the next item in the decoded frame queue. If so, it pops the item off the queue and displays it. Ideally a/v sync would take into account the actual audio clock of the audio hardware for the sync rather than using the system clock. Unfortunately getting valid time data out of the audio hardware has proven to be unreliable across platforms (and even distributions in Linux) depending on audio hardware, audio backend etc. The current approach works fine in practice and is a compromise until this issue can be sorted. The plan is to eventually move to synchronising using the audio hardware. To prevent audio skipping and framerate dropping it is very important to make sure no blocking occurs during the decoding process and minimise expensive time operations at the time a frame is to be displayed. This is managed by immediately converting video data to RGB on decode (an expensive operation to do at frame display time) and checking if the sound device will not block before writing sound data to it. Shutdown needs to ensure that the event posted to the decode thread is completed. The decode thread can potentially block internally inside liboggplay when reading, seeking, or its internal buffers containing decoded data are full. When blocked in this manner a call from the main thread to Shutdown() will hang. This is fixed with a protocol to ensure that the decode event cleanly completes. The nsMediaStream that the nsChannelReader uses has a Cancel() method. Calling this before Shutdown() will close any internal streams or listeners resulting in blocked i/o completing with an error, and all future i/o on the stream having an error. This causes the decode thread to exit and Shutdown() can occur. If the decode thread is seeking then the same Cancel() operation causes an error to be returned from the seek call to liboggplay which exits out of the seek operation, and stops the seek state running on the decode thread. If the decode thread is blocked due to internal decode buffers being full, it is unblocked during the shutdown process by calling oggplay_prepare_for_close. In practice the OggPlay internal buffer should never fill as we retrieve and process the frame immediately on decoding. The Shutdown method on nsOggDecoder can spin the event loop as it waits for threads to complete. Spinning the event loop is a bad thing to happen during certain times like destruction of the media element. To work around this the Shutdown method does nothing by queue an event to the main thread to perform the actual Shutdown. This way the shutdown can occur at a safe time. This means the owning object of a nsOggDecoder object *MUST* call Shutdown when destroying the nsOggDecoder object. */ #if !defined(nsOggDecoder_h_) #define nsOggDecoder_h_ #include "nsISupports.h" #include "nsCOMPtr.h" #include "nsIThread.h" #include "nsIChannel.h" #include "nsChannelReader.h" #include "nsIObserver.h" #include "nsIFrame.h" #include "nsAutoPtr.h" #include "nsSize.h" #include "prlog.h" #include "prmon.h" #include "gfxContext.h" #include "gfxRect.h" #include "oggplay/oggplay.h" #include "nsMediaDecoder.h" class nsAudioStream; class nsOggDecodeStateMachine; class nsOggDecoder : public nsMediaDecoder { friend class nsOggDecodeStateMachine; // ISupports NS_DECL_ISUPPORTS // nsIObserver NS_DECL_NSIOBSERVER public: // Enumeration for the valid play states (see mPlayState) enum PlayState { PLAY_STATE_START, PLAY_STATE_LOADING, PLAY_STATE_PAUSED, PLAY_STATE_PLAYING, PLAY_STATE_SEEKING, PLAY_STATE_ENDED, PLAY_STATE_SHUTDOWN }; nsOggDecoder(); ~nsOggDecoder(); PRBool Init(); // This method must be called by the owning object before that // object disposes of this decoder object. virtual void Shutdown(); virtual float GetCurrentTime(); virtual nsresult Load(nsIURI* aURI, nsIChannel* aChannel, nsIStreamListener **aListener); // Start playback of a video. 'Load' must have previously been // called. virtual nsresult Play(); // Stop playback of a video, and stop download of video stream. virtual void Stop(); // Seek to the time position in (seconds) from the start of the video. virtual nsresult Seek(float time); virtual nsresult PlaybackRateChanged(); virtual void Pause(); virtual float GetVolume(); virtual void SetVolume(float volume); virtual float GetDuration(); virtual void GetCurrentURI(nsIURI** aURI); virtual nsIPrincipal* GetCurrentPrincipal(); virtual void UpdateBytesDownloaded(PRUint64 aBytes); // Called when the video file has completed downloading. // Call on the main thread only. void ResourceLoaded(); // Call from any thread safely. Return PR_TRUE if we are currently // seeking in the media resource. virtual PRBool IsSeeking() const; // Get the size of the media file in bytes. Called on the main thread only. virtual void SetTotalBytes(PRInt64 aBytes); // Set a flag indicating whether seeking is supported virtual void SetSeekable(PRBool aSeekable); // Return PR_TRUE if seeking is supported. virtual PRBool GetSeekable(); protected: // Change to a new play state. This updates the mState variable and // notifies any thread blocking on this objects monitor of the // change. Can be called on any thread. void ChangeState(PlayState aState); // Returns the monitor for other threads to synchronise access to // state PRMonitor* GetMonitor() { return mMonitor; } // Return the current state. The caller must have entered the // monitor. PlayState GetState() { return mPlayState; } /****** * The following methods must only be called on the main * thread. ******/ // Called when the metadata from the Ogg file has been read. // Call on the main thread only. void MetadataLoaded(); // Called when the first frame has been loaded. // Call on the main thread only. void FirstFrameLoaded(); // Called when the video has completed playing. // Call on the main thread only. void PlaybackEnded(); // Return the current number of bytes loaded from the video file. // This is used for progress events. virtual PRUint64 GetBytesLoaded(); // Return the size of the video file in bytes. // This is used for progress events. virtual PRInt64 GetTotalBytes(); // Buffering of data has stopped. Inform the element on the main // thread. void BufferingStopped(); // Buffering of data has started. Inform the element on the main // thread. void BufferingStarted(); // Seeking has stopped. Inform the element on the main // thread. void SeekingStopped(); // Seeking has started. Inform the element on the main // thread. void SeekingStarted(); // Called when the backend has changed the current playback // position. It dispatches a timeupdate event and invalidates the frame. // This must be called on the main thread only. void PlaybackPositionChanged(); private: // Register/Unregister with Shutdown Observer. // Call on main thread only. void RegisterShutdownObserver(); void UnregisterShutdownObserver(); /****** * The following members should be accessed on the main thread only ******/ // Total number of bytes downloaded so far. PRUint64 mBytesDownloaded; // The URI of the current resource nsCOMPtr mURI; // Thread to handle decoding of Ogg data. nsCOMPtr mDecodeThread; // The current playback position of the media resource in units of // seconds. This is updated approximately at the framerate of the // video (if it is a video) or the callback period of the audio. // It is read and written from the main thread only. float mCurrentTime; // Volume that playback should start at. 0.0 = muted. 1.0 = full // volume. Readable/Writeable from the main thread. Read from the // audio thread when it is first started to get the initial volume // level. float mInitialVolume; // Position to seek to when the seek notification is received by the // decoding thread. Written by the main thread and read via the // decoding thread. Synchronised using mPlayStateMonitor. If the // value is negative then no seek has been requested. When a seek is // started this is reset to negative. float mRequestedSeekTime; // Size of the media file in bytes. Set on the first non-byte range // HTTP request from nsChannelToPipe Listener. Accessed on the // main thread only. PRInt64 mContentLength; // True if we are registered with the observer service for shutdown. PRPackedBool mNotifyOnShutdown; // True if the media resource is seekable (server supports byte range // requests). PRPackedBool mSeekable; /****** * The following member variables can be accessed from any thread. ******/ // The state machine object for handling the decoding via // oggplay. It is safe to call methods of this object from other // threads. Its internal data is synchronised on a monitor. The // lifetime of this object is after mPlayState is LOADING and before // mPlayState is SHUTDOWN. It is safe to access it during this // period. nsCOMPtr mDecodeStateMachine; // OggPlay object used to read data from a channel. Created on main // thread. Passed to liboggplay and the locking for multithreaded // access is handled by that library. Some methods are called from // the decoder thread, and the state machine for that thread keeps // a pointer to this reader. This is safe as the only methods called // are threadsafe (via the threadsafe nsMediaStream). nsChannelReader* mReader; // Monitor for detecting when the video play state changes. A call // to Wait on this monitor will block the thread until the next // state change. PRMonitor* mMonitor; // Set to one of the valid play states. It is protected by the // monitor mMonitor. This monitor must be acquired when reading or // writing the state. Any change to the state on the main thread // must call NotifyAll on the monitor so the decode thread can wake up. PlayState mPlayState; // The state to change to after a seek or load operation. It is // protected by the monitor mMonitor. This monitor must be acquired // when reading or writing the state. Any change to the state must // call NotifyAll on the monitor. PlayState mNextState; }; #endif