Commit Graph

340 Commits

Author SHA1 Message Date
Philipp von Weitershausen
52ed8e305a Bug 609421 - Move tracker and store implementation into engines.js. r=rnewman 2011-01-18 16:23:25 -08:00
Philipp von Weitershausen
9ee1e6ed53 Bug 623813 - Preserve stack traces when rethrowing exceptions conditionally. r=mconnor 2011-01-13 14:01:05 -08:00
Richard Newman
906e72b628 Bug 618341: Rethrow exceptions we aren't supposed to handle. r=philiKON 2010-12-10 15:43:12 -08:00
Richard Newman
b700f4e4a5 Bug 617709: Tests and functionality for on-the-fly HMAC error recovery. r=philiKON 2010-12-09 23:06:44 -08:00
Richard Newman
98051adf04 Bug 616265: Add download limit for history, lift magic number. r=mconnor 2010-12-06 17:25:35 -08:00
Richard Newman
ffb07cee37 Bug 603489, 614489: simplified crypto. r=mconnor 2010-11-29 16:41:17 -08:00
Philipp von Weitershausen
31d6d2b3c1 Bug 591126 - Handle upload interruption gracefully [r=mconnor] 2010-11-23 21:21:31 -08:00
Philipp von Weitershausen
7440175181 Bug 610923 - Allow engines to bypass the tracker for certain or all changed items [r=mconnor]
SyncEngine now uses the getChangedIDs() method to get all changed IDs before a sync. By default this still goes through the tracker, but engines can override this method to do queries based on timestamps, or to bypass the tracker entirely.

SyncEngine now also exposes a lastSyncLocal property that keeps the local timestamp of the last sync so that engines can do timestamp based queries.
2010-11-11 11:00:35 -08:00
Philipp von Weitershausen
08efe3cb23 Bug 609398 - Get rid of partial sync [r=mconnor] 2010-11-09 13:51:19 -08:00
Philipp von Weitershausen
3065b1ede2 Bug 601973 - SyncEngine._testDecrypt() yields wrong result [r=mconnor]
Fix a 'this' scoping error in SyncEngine._testDecrypt(). Rename this method to canDecrypt() since it's clearly public API.

Provide tests for SyncEngine.canDecrypt() as well as Service.wipeClient().
2010-10-05 20:32:56 +02:00
Philipp von Weitershausen
753f9d8d3e Bug 600995 - Use a record's "encryption" property only as a fallback, default to the engine's value. [r=mconnor] 2010-10-04 22:39:08 +02:00
Philipp von Weitershausen
6e250f858f Bug 569968 - Migration requires client to remove absolute URLs in crypto records
Make WBORecord.uri, CryptoWrapper.encryption, PubKey.privateKeyUri, PrivKey.publicKeyUri as well as the CryptoMeta keyring keys relative URLs in the payload.  As a result, all records now *must* know their URI.

Bump storage version from 2 to 3 to trigger reupload of all data and exclude older clients.
2010-08-12 22:19:39 +02:00
Philipp von Weitershausen
c29a0e55af Bug 578671 - Sync which engines are enabled across clients, wipe data for disabled engines [r=mconnor] 2010-09-07 18:44:01 +02:00
Philipp von Weitershausen
a2466ae2e3 Bug 582083 - Should inspect POST responses for failed WBOs [r=Mardak]
Make sure records that failed to upload continue to be marked in the tracker so that they'll be uploaded again in the next sync.
2010-07-31 13:28:00 +02:00
Mike Connor
39e9fff92b Bug 578825 - sync should only sort on mobile clients, r=Mardak 2010-07-14 18:18:24 -04:00
Edward Lee
4e55362a90 Bug 570636 - Decide how to co-exist as a sync add-on and built-in sync [r=mconnor]
Map the modules directory to services-sync instead of weave and update imports.
2010-06-16 14:30:08 -07:00
Edward Lee
0d25dd2924 Bug 568677 - Failure to get CryptoMeta assumes it's missing [r=mconnor]
Explicitly check the status code to make sure we stop syncing on non-404 crypto meta failures. Add tests to check correct behavior of 404 and non-404 errors during syncStartup.
2010-06-14 15:16:53 -07:00
Edward Lee
9ddf0f8ed5 Bug 546551 - Weave status bar notifications are too busy [r=mconnor]
Remove status bar icon/text for setup/connect/sync status. Add menuitem for setting up sync instead of drilling down a 1-item submenu that leads to a button. Remove unused Engine.displayName and related strings.
2010-06-14 10:30:37 -07:00
Philipp von Weitershausen
721e3fa637 Bug 557596 - code audit and create unit test plan for resource.js [r=mconnor]
Lots of resource tests, Resource.serverTime initialized to null, Resource.headers normalized to lowercase.
2010-06-01 15:15:53 -07:00
Philipp von Weitershausen
ceaa8fab03 Bug 557588 - code audit and create unit test plan for engines.js [r=mconnor]
Tests for SyncEngine.sync(), incl some additions to harness.
2010-06-01 15:07:50 -07:00
Philipp von Weitershausen
ba3ea98599 Bug 557588 - code audit and create unit test plan for engines.js [r=mconnor]
Tests for EngineMangerSvc, Engine and SyncEngine sans sync().
2010-06-01 15:06:16 -07:00
Edward Lee
98d3a793fe Bug 567371 - replace server and replace local options does not sync certain Passwords/bookmarks [r=mconnor]
Make sure to clear local cache when deleting crypto records from the server. Handle missing crypto by deleting any existing data and reuploading. Fix broken records by uploading new ones.
2010-05-21 12:15:58 -07:00
Edward Lee
433ac3095f Bug 562515 - Reset sync option "Replace all data on this computer with your Weave data" is not handling local deletes correctly [r=mconnor]
Clear out any changed ids when wiping the client to prevent delete records from getting uploaded for these items.
2010-05-03 14:39:32 -07:00
Edward Lee
210f158298 Bug 556454 - engine.sync should always check engine.enabled [r=mconnor]
Just check if the engine is enabled at the beginning of sync. For now keep getEnabled as it's used for other behavior and not just sync.
2010-04-29 16:42:21 -07:00
Mike Connor
914768c99b bug 496485 - make Service.wipeServer work properly, r=Mardak 2010-04-07 20:06:37 -04:00
Edward Lee
32b2f2a15a Bug 557891 - Wipe local triggers deletions on all other clients! [r=mconnor]
Ignore tracker changes triggered by wiping the local client store.
2010-04-07 17:52:22 -07:00
Edward Lee
3b4aa0220d Bug 550627 - Default reconciliation to server wins for older changed items [r=mconnor]
Save the time the tracker adds a new changed id and use that to compare the age of the record on the server vs the age of the local change to decide if it's server wins or client wins. Fix up various direct uses of changedIDs to use the API and make the save-to-disk lazy to avoid excessive writes. Add a test to make sure addChangedID only increases in time.
2010-04-01 15:54:53 -07:00
Edward Lee
e2f2dee6c3 Bug 546768 - form history uses hashes for GUIDs [r=mconnor]
Write a FormWrapper that knows about GUIDs and get/sets them in moz_formhistory as needed. It lazily adds the columns on failure and lazily generates GUIDs for entries that are missing it. Don't eagerly create a sha1 formItem mapping -- don't create it at all, so empty syncs will be much faster too.
2010-04-01 15:29:16 -07:00
Edward Lee
0ad782b0b5 Bug 503964 - Have per-engine versioning to avoid wiping all engine data [r=mconnor]
Add an engines object to meta/global to track version and syncID for each engine. If the server is outdated, wipe the data and set a new version and syncID. If the client is oudated, ask for an upgrade. Differing syncIDs cause a reupload. All engines are right now the default version 1.
2010-03-25 10:05:21 -07:00
Edward Lee
e73dbf06b9 Bug 547007 - Use a per-record IV instead of one for each symkey [r=mconnor]
Generate a random IV on every encrypt instead of taking it from the CryptoMeta. Don't bother generating a bulkIV per CryptoMeta.
2010-03-16 16:31:56 -07:00
Edward Lee
177b122960 Bug 547048 - Only allow clients to issue delete records [r=mconnor]
Don't specially serialize/not encrypt delete records and store the deleted flag as part of the cleartext payload.
2010-03-16 16:31:56 -07:00
Edward Lee
00f548581e Bug 544069 - Move bookmark parentid into the encrypted payload [r=mconnor]
Tweak the get/setters to refer to the PlacesItem instead of WBORecord.
2010-03-16 16:31:56 -07:00
Edward Lee
accbff6fd1 Bug 551874 - Figure out if we can avoid getting into partial synced state [r=mconnor]
For not-mobile clients, sync everything by specifying Infinity to keep existing math/logic working. Don't add a limit ?query if it's Infinity.
2010-03-19 11:35:01 -07:00
Edward Lee
1ce05d4603 Bug 549633 - Standardize record creation so all SyncEngines encrypt data [r=mconnor]
Always add the id and encryption value so client and delete records can be encrypted.
2010-03-05 14:46:48 -08:00
Edward Lee
a9c2e6c4c2 Bug 549632 - Remove storage cache, which is mostly un/incorrectly used [r=mconnor]
Remove incorrectly used cache from some engines and clean up references from SyncEngine.
2010-03-05 14:43:11 -08:00
Edward Lee
1ad29052b7 Bug 545767 - Clean up various names/strings needed for each set of engines/stores/trackers
Pass the engine name when constructing a subclass engine and construct the store/tracker with the same name.
2010-02-11 15:29:15 -08:00
Edward Lee
b8de893593 Bug 545764 - Convert constructor -> _init pattern to just use constructor
Inline various _init calls and do super's init with <Super>.call(this, args..). Add various get/set sugar to those missing e.g., meta.keyring. Also simplify crypto record creation by setting cleartext in the parent.

--HG--
extra : rebase_source : 6c9a9f210f8f46ac338adb84188538e7353c9673
2010-02-11 15:25:31 -08:00
Edward Lee
e8ccb18a12 Bug 536595 - Sync fewer items per data type on mobile
Just sync a flat 50 instead of .1 of 1500 to avoid increasing slowdown when processing many items.
2010-01-06 10:00:05 -08:00
Edward Lee
def7a2495e Bug 536594 - Warn on record creation failure but continue the sync
Wrap createRecord and encrypt incase an engine fails to create one or more of the records similar to how process incoming records are wrapped.
2010-01-06 09:59:05 -08:00
Edward Lee
fdaee13190 Bug 535722 - Reduce the amount of logging for default log levels
Include the URI on success/fail requests and only trace log the onStartRequest. Switch various debug messages to trace and remove importing Log4Moz in fx-weave-overlay and generic-change. Drop the rootLogger to Debug to not log trace messages from unpreffed loggers.
2009-12-17 18:51:55 -08:00
Edward Lee
86d2294e4d Bug 531005 - Sync is not working after update IWeaveCrypto.unwrapSymmetricKey NS_ERROR_FAILURE
Check that the cryptometa is unwrappable when syncing; if not purge the key and data and make a new crypto record.
2009-12-02 17:25:14 -08:00
Edward Lee
eb87765d7b Bug 530823 - Engines need to be able to specify a prefName for sharing prefs (different history types)
Use prefName for checking if an engine is enabled and have forms share a prefName with history.
2009-11-30 12:15:18 -08:00
Edward Lee
d4295ef8e9 Remove unused prefs and imports. (Bug 524916)
--HG--
extra : rebase_source : 074e542f63ee840b61db7fe63a6c3540c13a31a5
2009-11-25 17:49:15 -08:00
Edward Lee
c05e0c5f94 Remove trailing whitespace from the codebase. 2009-11-20 14:34:20 -08:00
Edward Lee
8bcc3584b3 Bug 526942 - Try to sync tabs when viewing remote tabs
Try fetching tabs when loading the remote tabs view and if the sync got new tabs, reload the page.
2009-11-12 15:18:43 -08:00
Edward Lee
89460e2378 Bug 527766 - Sync history pages with a single visit
Backout bug 518972 (6954c93b8903) so now we push all data instead of pages with more than 1 visit.
2009-11-10 15:52:40 -08:00
Edward Lee
5013b211cb Bug 527775 - Allow customizable api version on the client
Add a new pref storageAPI (currently 0.5) that gets used for storage urls.

--HG--
extra : rebase_source : 74ca13de918d4211cbdcf708c30cc03d876fec7d
2009-11-10 15:24:31 -08:00
Edward Lee
6c926b2c49 Avoid undefined property warnings by checking for null first instead of comparing. 2009-11-04 15:12:29 -08:00
Mike Connor
bcb7b750cb bug 485108 - drop status dialog, show status including engines on statusbar, remove dead status bits 2009-10-28 03:42:35 -04:00
Edward Lee
862fb7381e Ignore sortindex when comparing if records are different because it's only used for ordering of records. 2009-10-16 16:18:38 -07:00
Edward Lee
bd2a51d362 Bug 522077 - Download a certain total number of items each sync instead of fixed "catch up"
Limit the initial the first fetch of new items by a total number of fetch and subtract the number of items processed. Use the difference to keep fetching more items from the backlog in chunks.
2009-10-13 11:20:28 -07:00
Edward Lee
8ce9ab3899 Split multiple id deletes into 100-id chunks instead of hitting max apache URI length. 2009-10-12 18:11:31 -07:00
Edward Lee
13cddbf9bc Bug 520215 - Only wipe if data is able to be restored from the server
For each engine, try decrypting a record from the server before wiping local data.
2009-10-12 16:22:54 -07:00
Edward Lee
a994e4258f Simplify logic for Engines.getAll/Enabled and use Enabled for main sync loop. 2009-10-08 13:51:22 -07:00
Edward Lee
32f373dcc0 Don't unconditionally forceGC on processIncoming especially when there's nothing to process. 2009-09-25 18:39:24 -07:00
Edward Lee
a69c3b332a Sort the sync timing report to show certain functions first. 2009-09-25 18:38:47 -07:00
Edward Lee
4f389c5c8a Bug 518972 - Only upload history records that have more than one visit
Allow each engine to provide a custom Collection object and have History provide a collection that filters out certain data. This is inefficient because we have to first create then encrypt the record before we can filter it out.
2009-09-25 16:52:12 -07:00
Edward Lee
eba12cbe8d Bug 518018 - Default to serverURL instead of userAPI on 404 cluster check
Rework server/user/misc prefs to allow relative paths and full urls for generating API paths. Cache string properties of generated URLs under the storageAPI instead of using dynamic getters.
2009-09-21 17:13:41 -07:00
Edward Lee
8db1de49fa Merge weave to about-weave. 2009-09-16 19:16:48 -07:00
Edward Lee
b48bdeef35 Remove meta serialization now that resource takes objects. 2009-09-16 19:15:55 -07:00
Dan Mills
5094a04f3d initial patch
--HG--
extra : rebase_source : 250c8f0d85483300147a6f0d691e0214db6d837b
2009-09-15 21:54:05 -04:00
Mike Connor
4faaba2018 bug 511549 - make detailedStatus much smarter about errors and backoff, r=edilee
--HG--
extra : rebase_source : 42e3eb9c89d2567d18c02e7583202e284dd69b0e
2009-09-15 21:38:52 -04:00
Edward Lee
91e977a052 Fix exception name. 2009-09-11 13:10:19 -07:00
Edward Lee
e856402be7 Add some comments to partial download code and remove other comments. 2009-09-11 08:24:42 -07:00
Edward Lee
9380981a9e Merge weave to weave-partial-view. 2009-09-10 23:18:33 -07:00
Edward Lee
fb1bc5da49 Bug 507429 - Partial sync download support
Only fetch a limited number of items on first/update syncs and if we get the same number, ask the server for the ids to fetch later. Also on every download, process some of the backlog and save the list of GUIDs to disk as json for cross-session support.
2009-09-10 23:11:33 -07:00
Edward Lee
8b8447ab6e Remove some unnecessary logs that follow right after the notify/event now that we only show verbose. Only log non-success response body for Trace. 2009-09-10 22:57:36 -07:00
Edward Lee
b59b775794 Merge weave to weave-partial-view. 2009-09-10 12:42:29 -07:00
Edward Lee
8f5372e6b1 Update lastSync to the lastModified time from info/collections instead of updating times on each record. This is to prepare for fetching a list of GUIDs and processing a list of GUIDs. 2009-09-10 12:41:38 -07:00
Edward Lee
b0c3b9005e Bug 515676 - Allow giving objects to PUT/POST without Filters to avoid stringify([parse(stringify(stringify(obj)))])
Get rid of Filters and automatically JSON.stringify PUT/POST data that aren't strings, so plain Records can be passed in to PUT and POST. This leverages toJSON of Records to provide an object that can be serialized. Fix up client record serialize/deserialize to still escape/unescape non-ASCII.
2009-09-10 11:05:13 -07:00
Edward Lee
0dc7cf0d81 Merge weave to weave-partial-view. 2009-09-08 23:35:21 -07:00
Edward Lee
258a985cee Expose a _handleDupe on engines and provide a custom one for bookmarks that tracks GUID changes so that it can keep an alias mapping to fix incoming item properties (id, parent, predecessor). Move out _reparentOrphans so that it is triggered on update and not just create because folders can change ids to the right parent. 2009-09-08 23:33:15 -07:00
Edward Lee
19443b737b Bug 514323 - Use info/collections to check if there's new data. r=thunder
Fetch info/collections before syncing engines and set the lastModified property on the engine so that they can check if they need to process incoming.
2009-09-03 21:30:40 -07:00
Edward Lee
9ec91eed07 Bug 514600 - Track how long various parts of sync take
Instrument all functions that are part of the sync engine (except some constructors, etc.) and generate statistics (min/max/sum/num/avg) for processing. For now with the default appender, implement toString to report just the total time.
2009-09-03 20:11:36 -07:00
Edward Lee
0fa22bedac Switch to 0.5 server API for deleting multiple ids from a collection. 2009-08-31 18:30:44 -07:00
Edward Lee
5a3c53fd6a Merge weave to weave-partial-view w/ conflicts. 2009-08-31 18:03:35 -07:00
Edward Lee
f8541ddf2f Pick a canonical guid to keep in a way that all machines can agree without having all data -- "smaller" guid. 2009-08-31 17:51:26 -07:00
Edward Lee
994b05a7d1 Implement multiple ids deletion for use with 0.3 APIs. 2009-08-31 17:50:23 -07:00
Edward Lee
97e3f746cb Switch end-of-sync deletes to something SyncEngine is aware of and runs at _syncFinish. 2009-08-31 17:30:34 -07:00
Edward Lee
5468320159 Convert uses of recordLike/findLikeId to findDupe except Bookmarks is unimplemented. 2009-08-31 17:27:30 -07:00
Edward Lee
4175f5ee7e Resolve lastChannel removal and 0.5 api changes. r=thunder 2009-08-26 16:09:48 -07:00
Edward Lee
9fd47b7f98 Merge weave to weave-0.5-api w/ conflicts. 2009-08-26 15:42:05 -07:00
Edward Lee
009411f482 Bug 511746 - Resource.foo shouldn't throw except in exceptional cases. r=thunder
Get rid of lastChannel and return a String object from _request with additional properties of status, succeeded, headers -- even if the response was handled by cache. Update engines to check for non-success and throw the failure. Update tests to use these additional properties instead of lastChannel, etc.
2009-08-26 15:32:46 -07:00
Edward Lee
ca6fe1108c Backed out changeset 129ca9a54aed due to burning
test_auth_manager:      FAIL
test_resource:  FAIL
2009-08-26 01:50:36 -07:00
Anant Narayanan
371e639357 Change Resource.get() semantics and support X-Weave-Alert (bug #478330)
--HG--
extra : rebase_source : e0eb1e34f85ddd041005d780590640005dc0b434
2009-08-26 00:05:57 -07:00
Edward Lee
8431f6ff7f Read out the timestamp of the POST response header because the modified time is not part of the response data. 2009-08-25 18:04:46 -07:00
Anant Narayanan
e7097f319f Bug 507433 - Update client to use the weave 0.5 server. r-Mardak
Initial 0.5 server API switch not quite working.
2009-08-25 17:06:13 -07:00
Edward Lee
dfe8e7e513 Don't include changes to special places Library "exclude from backup" items. 2009-08-25 16:15:05 -07:00
Edward Lee
e6732d5263 Conditionally log trace records to avoid always doing record.toString().
--HG--
extra : rebase_source : 65d8025324ed95eac29d530e2699a2f29c8a2cb7
2009-08-20 17:00:15 -07:00
Edward Lee
014c01848d Print out the stack when failing to apply an incoming record. 2009-08-16 12:39:23 -07:00
Edward Lee
2335b057db Get rid of depth and sort on index instead. 2009-08-15 00:56:27 -07:00
Edward Lee
3ca750f464 Remove meta/mini records. Bye! 2009-08-13 18:50:54 -07:00
Edward Lee
6ddbaa9a8e Bug 506795 - Handle adding duplicate logins. r=thunder
Refactor reconcile to call findLikeId that by default will look for recordLike in the outgoing queue as it does now. Override findLikeId for password engine to search local logins.
2009-07-28 10:06:02 -07:00
Edward Lee
551f097468 Convert various debug messages to trace. 2009-07-22 21:40:18 -07:00
Dan Mills
db7966a67b Bug 505906: Don't do a last POST if there is nothing left to send. Regression from bug 481347. r=mardak 2009-07-22 17:48:47 -07:00
Edward Lee
a26f0b1166 Bug 485573 - Fennec runs out of memory and crashes if too many history items to sync. r=thunder
Incrementally process records as the collection finds record boundaries and converts them to records for the engine to use. Get rid of the collection iterator and original RecordParser. Add tests for incremental record parsing and remove old iter tests.
2009-07-22 16:38:34 -07:00
Dan Mills
7f9996ac73 Bug 481347: Split uploads into multiple chunks to get around server upload limit and curtail memory usage 2009-07-22 16:21:33 -07:00
Edward Lee
baa48207f2 Bug 504387 - Cache keys after creating/PUTting them instead of after GETting. r=thunder
Set the record manager for CryptoMetas, PubKeys, PrivKeys after uploading keys.
2009-07-16 23:13:58 -07:00
Edward Lee
68d361380a Bug 504230 - Use Engine's logging level for Store and Tracker
Use the log.logger.engine.* prefs to determine what the Store and Tracker should log with. This requires putting the same engine name on each Store/Tracker object, so there's some duplicate string values right now.
2009-07-14 18:34:03 -07:00
Edward Lee
e94143f4e2 Bug 504212 - Have javascript stack traces show [object Object] for various CryptoWrapper functions
Pass the ID object instead of reading out the password and passing it in to encrypt/decrypt.
2009-07-14 16:51:04 -07:00