32 Commits

Author	SHA1	Message	Date
Berker Peksag	9a7bbb2e3f	Issue #25400: RobotFileParser now correctly returns default values for crawl_delay and request_rate ... Initial patch by Peter Wirtz.	2016-09-18 20:17:58 +03:00
Berker Peksag	2a8d7f1c47	Issue #28151: Use pythontest.net in test_robotparser	2016-09-18 11:21:57 +03:00
Berker Peksag	a3c1728bb6	Use HTTP in testPythonOrg	2016-09-11 15:46:47 +03:00
Berker Peksag	966ad74bf9	Unskip testPythonOrg in test_robotparser ... We should probably use pythontest.net for this.	2016-09-11 15:27:07 +03:00
Berker Peksag	2a9f5edeeb	Wrap testPasswordProtectedSite with @reap_threads	2016-09-11 15:17:53 +03:00
Berker Peksag	4da0fd06ce	Issue #25497: Rewrite test_robotparser to use a class based design	2016-09-11 14:53:16 +03:00
Serhiy Storchaka	e437a10d15	Issue #23277: Remove unused imports in tests.	2016-04-24 21:41:02 +03:00
Berker Peksag	960e848f0d	Issue #16099: RobotFileParser now supports Crawl-delay and Request-rate ... extensions. Patch by Nikolay Bogoychev.	2015-10-08 12:27:06 +03:00
Berker Peksag	ad324f6bcc	Issue #20753: Skip PasswordProtectedSiteTestCase when Python is built without threads.	2014-06-29 15:54:56 +03:00
Senthil Kumaran	601d6ec693	issue20753 - robotparser tests should not rely upon external resource when not required. ... Specifically, it was relying a URL which gave HTTP 403 and used it to assert it's methods, this changes undoes that and provides a local http server with similar properties. Patch contributed by Vajrasky Kok.	2014-06-25 02:58:15 -07:00
Zachary Ware	66f2928479	Issue #18492: Allow all resources when tests are not run by regrtest.py. ... This changeset also includes cleanup allowed by this behavior change.	2014-06-02 16:01:29 -05:00
Georg Brandl	89e5671be7	#20719: Disable the robotparser python.org test until the gzip encoding issue can be sorted.	2014-02-23 08:45:15 +01:00
Senthil Kumaran	c70a6ae49b	#17403: urllib.parse.robotparser normalizes the urls before adding to ruleline. ... This helps in handling certain types invalid urls in a conservative manner.	2013-05-29 05:54:31 -07:00
Ezio Melotti	0fb37ea34d	#17066: test_robotparser now works with unittest test discovery. Patch by Zachary Ware.	2013-03-12 07:49:12 +02:00
Antoine Pitrou	95531ea2f1	Avoid failing in test_robotparser when mueblesmoraleda.com is flaky and ... an overzealous DNS service (e.g. OpenDNS) redirects to a placeholder Web site.	2011-07-08 19:43:51 +02:00
Antoine Pitrou	8bc09039ed	Improve transient_internet() again to detect more network errors, ... and use it in test_robotparser. Fixes #8574.	2010-09-07 21:09:09 +00:00
Georg Brandl	0a0fc07d37	#4108: the first default entry (User-agent: *) wins.	2010-07-29 17:55:01 +00:00
Senthil Kumaran	3f8ab965f7	Fix Issue6325 - robotparse to honor urls with query strings.	2010-07-28 16:27:56 +00:00
Florent Xicluna	41fe615539	(partially) ... Merged revisions 79534,79537,79539,79558,79606 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r79534 | florent.xicluna | 2010-03-31 23:21:54 +0200 (mer, 31 mar 2010) | 2 lines Fix test for xml.etree when using a non-ascii path. And use check_warnings instead of catch_warnings. ........ r79537 | florent.xicluna | 2010-03-31 23:40:32 +0200 (mer, 31 mar 2010) | 2 lines Fix typo ........ r79539 | florent.xicluna | 2010-04-01 00:01:03 +0200 (jeu, 01 avr 2010) | 2 lines Replace catch_warnings with check_warnings when it makes sense. Use assertRaises context manager to simplify some tests. ........ r79558 | florent.xicluna | 2010-04-01 20:17:09 +0200 (jeu, 01 avr 2010) | 2 lines #7092: Fix some -3 warnings, and fix Lib/platform.py when the path contains a double-quote. ........ r79606 | florent.xicluna | 2010-04-02 19:26:42 +0200 (ven, 02 avr 2010) | 2 lines Backport some robotparser test and skip the test if the external resource is not available. ........	2010-04-02 18:52:12 +00:00
Antoine Pitrou	1bfd0cc5d4	Furniture is not very reliable these days (buildbot failures).	2010-04-02 17:12:12 +00:00
Benjamin Peterson	c9c0f201fe	convert old fail* assertions to assert*	2009-06-30 23:06:06 +00:00
Benjamin Peterson	d63137159b	Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from ... svn+ssh://pythondev@svn.python.org/python/trunk ........ r65209 | raymond.hettinger | 2008-07-23 19:08:18 -0500 (Wed, 23 Jul 2008) | 1 line Finish-up the partial conversion from int to Py_ssize_t for deque indices and length. ........ r65210 | raymond.hettinger | 2008-07-23 19:53:49 -0500 (Wed, 23 Jul 2008) | 1 line Parse to the correct datatype. ........ r65211 | benjamin.peterson | 2008-07-23 21:27:46 -0500 (Wed, 23 Jul 2008) | 1 line fix spacing ........ r65212 | benjamin.peterson | 2008-07-23 21:31:28 -0500 (Wed, 23 Jul 2008) | 1 line fix markup ........ r65213 | benjamin.peterson | 2008-07-23 21:45:37 -0500 (Wed, 23 Jul 2008) | 1 line add some documentation for 2to3 ........ r65214 | raymond.hettinger | 2008-07-24 00:38:48 -0500 (Thu, 24 Jul 2008) | 1 line Finish conversion from int to Py_ssize_t. ........ r65215 | raymond.hettinger | 2008-07-24 02:04:55 -0500 (Thu, 24 Jul 2008) | 1 line Convert from long to Py_ssize_t. ........ r65216 | georg.brandl | 2008-07-24 02:09:21 -0500 (Thu, 24 Jul 2008) | 2 lines Fix indentation. ........ r65225 | benjamin.peterson | 2008-07-25 11:55:37 -0500 (Fri, 25 Jul 2008) | 1 line teach .bzrignore about doc tools ........ r65226 | benjamin.peterson | 2008-07-25 12:02:11 -0500 (Fri, 25 Jul 2008) | 1 line document default value for fillvalue ........ r65233 | raymond.hettinger | 2008-07-25 13:43:33 -0500 (Fri, 25 Jul 2008) | 1 line Issue 1592: Better error reporting for operations on closed shelves. ........ r65239 | benjamin.peterson | 2008-07-25 16:59:53 -0500 (Fri, 25 Jul 2008) | 1 line fix indentation ........ r65246 | andrew.kuchling | 2008-07-26 08:08:19 -0500 (Sat, 26 Jul 2008) | 1 line This sentence continues to bug me; rewrite it for the second time ........ r65247 | andrew.kuchling | 2008-07-26 08:09:06 -0500 (Sat, 26 Jul 2008) | 1 line Remove extra words ........ r65255 | skip.montanaro | 2008-07-26 19:49:02 -0500 (Sat, 26 Jul 2008) | 3 lines Close issue 3437 - missing state change when Allow lines are processed. Adds test cases which use Allow: as well. ........ r65256 | skip.montanaro | 2008-07-26 19:50:41 -0500 (Sat, 26 Jul 2008) | 2 lines note robotparser bug fix. ........	2008-07-31 16:23:04 +00:00
Georg Brandl	3dbca81c9b	Merged revisions 65012,65035,65037-65040,65048,65057,65077,65091-65095,65097-65099,65127-65128,65131,65133-65136,65139,65149-65151,65155,65158-65159,65176-65178,65183-65184,65187-65190,65192,65194 via svnmerge from ... svn+ssh://pythondev@svn.python.org/python/trunk ........ r65012 | jesse.noller | 2008-07-16 15:24:06 +0200 (Wed, 16 Jul 2008) | 2 lines Apply patch for issue 3090: ARCHFLAGS parsing incorrect ........ r65035 | georg.brandl | 2008-07-16 23:19:28 +0200 (Wed, 16 Jul 2008) | 2 lines #3045: fix pydoc behavior for TEMP path with spaces. ........ r65037 | georg.brandl | 2008-07-16 23:31:41 +0200 (Wed, 16 Jul 2008) | 2 lines #1608818: errno can get set by every call to readdir(). ........ r65038 | georg.brandl | 2008-07-17 00:04:20 +0200 (Thu, 17 Jul 2008) | 2 lines #3305: self->stream can be NULL. ........ r65039 | georg.brandl | 2008-07-17 00:09:17 +0200 (Thu, 17 Jul 2008) | 2 lines #3345: fix docstring. ........ r65040 | georg.brandl | 2008-07-17 00:33:18 +0200 (Thu, 17 Jul 2008) | 2 lines #3312: fix two sqlite3 crashes. ........ r65048 | georg.brandl | 2008-07-17 01:35:54 +0200 (Thu, 17 Jul 2008) | 2 lines #3388: add a paragraph about using "with" for file objects. ........ r65057 | gregory.p.smith | 2008-07-17 05:13:05 +0200 (Thu, 17 Jul 2008) | 2 lines news note for r63052 ........ r65077 | jesse.noller | 2008-07-17 23:01:05 +0200 (Thu, 17 Jul 2008) | 3 lines Fix issue 3395, update _debugInfo to be _debug_info ........ r65091 | ronald.oussoren | 2008-07-18 07:48:03 +0200 (Fri, 18 Jul 2008) | 2 lines Last bit of a fix for issue3381 (addon for my patch in r65061) ........ r65092 | vinay.sajip | 2008-07-18 10:59:06 +0200 (Fri, 18 Jul 2008) | 1 line Issue #3389: Allow resolving dotted names for handlers in logging configuration files. Thanks to Philip Jenvey for the patch. ........ r65093 | vinay.sajip | 2008-07-18 11:00:00 +0200 (Fri, 18 Jul 2008) | 1 line Issue #3389: Allow resolving dotted names for handlers in logging configuration files. Thanks to Philip Jenvey for the patch. ........ r65094 | vinay.sajip | 2008-07-18 11:00:35 +0200 (Fri, 18 Jul 2008) | 1 line Issue #3389: Allow resolving dotted names for handlers in logging configuration files. Thanks to Philip Jenvey for the patch. ........ r65095 | vinay.sajip | 2008-07-18 11:01:10 +0200 (Fri, 18 Jul 2008) | 1 line Issue #3389: Allow resolving dotted names for handlers in logging configuration files. Thanks to Philip Jenvey for the patch. ........ r65097 | georg.brandl | 2008-07-18 12:20:59 +0200 (Fri, 18 Jul 2008) | 2 lines Remove duplicate entry in __all__. ........ r65098 | georg.brandl | 2008-07-18 12:29:30 +0200 (Fri, 18 Jul 2008) | 2 lines Correct attribute name. ........ r65099 | georg.brandl | 2008-07-18 13:15:06 +0200 (Fri, 18 Jul 2008) | 3 lines Document the different meaning of precision for {:f} and {:g}. Also document how inf and nan are formatted. #3404. ........ r65127 | raymond.hettinger | 2008-07-19 02:42:03 +0200 (Sat, 19 Jul 2008) | 1 line Improve accuracy of gamma test function ........ r65128 | raymond.hettinger | 2008-07-19 02:43:00 +0200 (Sat, 19 Jul 2008) | 1 line Add recipe to the itertools docs. ........ r65131 | georg.brandl | 2008-07-19 12:08:55 +0200 (Sat, 19 Jul 2008) | 2 lines #3378: in case of no memory, don't leak even more memory. :) ........ r65133 | georg.brandl | 2008-07-19 14:39:10 +0200 (Sat, 19 Jul 2008) | 3 lines #3302: fix segfaults when passing None for arguments that can't be NULL for the C functions. ........ r65134 | georg.brandl | 2008-07-19 14:46:12 +0200 (Sat, 19 Jul 2008) | 2 lines #3303: fix crash with invalid Py_DECREF in strcoll(). ........ r65135 | georg.brandl | 2008-07-19 15:00:22 +0200 (Sat, 19 Jul 2008) | 3 lines #3319: don't raise ZeroDivisionError if number of rounds is so low that benchtime is zero. ........ r65136 | georg.brandl | 2008-07-19 15:09:42 +0200 (Sat, 19 Jul 2008) | 3 lines #3323: mention that if inheriting from a class without __slots__, the subclass will have a __dict__ available too. ........ r65139 | georg.brandl | 2008-07-19 15:48:44 +0200 (Sat, 19 Jul 2008) | 2 lines Add ordering info for findall and finditer. ........ r65149 | raymond.hettinger | 2008-07-20 01:21:57 +0200 (Sun, 20 Jul 2008) | 1 line Fix compress() recipe in docs to use itertools. ........ r65150 | raymond.hettinger | 2008-07-20 01:58:47 +0200 (Sun, 20 Jul 2008) | 1 line Clean-up itertools docs and recipes. ........ r65151 | gregory.p.smith | 2008-07-20 02:22:08 +0200 (Sun, 20 Jul 2008) | 9 lines fix issue3120 - don't truncate handles on 64-bit Windows. This is still messy, realistically PC/_subprocess.c should never cast pointers to python numbers and back at all. I don't have a 64-bit windows build environment because microsoft apparently thinks that should cost money. Time to watch the buildbots. It builds and passes tests on 32-bit windows. ........ r65155 | georg.brandl | 2008-07-20 13:50:29 +0200 (Sun, 20 Jul 2008) | 2 lines #926501: add info where to put the docstring. ........ r65158 | neal.norwitz | 2008-07-20 21:35:23 +0200 (Sun, 20 Jul 2008) | 1 line Fix a couple of names in error messages that were wrong ........ r65159 | neal.norwitz | 2008-07-20 22:39:36 +0200 (Sun, 20 Jul 2008) | 1 line Fix misspeeld method name (negative) ........ r65176 | amaury.forgeotdarc | 2008-07-21 23:36:24 +0200 (Mon, 21 Jul 2008) | 4 lines Increment version number in NEWS file, and move items that were added after 2.6b2. (I thought there was a script to automate this kind of updates) ........ r65177 | amaury.forgeotdarc | 2008-07-22 00:00:38 +0200 (Tue, 22 Jul 2008) | 5 lines Issue2378: pdb would delete free variables when stepping into a class statement. The problem was introduced by r53954, the correction is to restore the symmetry between PyFrame_FastToLocals and PyFrame_LocalsToFast ........ r65178 | benjamin.peterson | 2008-07-22 00:05:34 +0200 (Tue, 22 Jul 2008) | 1 line don't use assert statement ........ r65183 | ronald.oussoren | 2008-07-22 09:06:00 +0200 (Tue, 22 Jul 2008) | 2 lines Fix buglet in fix for issue3381 ........ r65184 | ronald.oussoren | 2008-07-22 09:06:33 +0200 (Tue, 22 Jul 2008) | 2 lines Fix build issue on OSX 10.4, somehow this wasn't committed before. ........ r65187 | raymond.hettinger | 2008-07-22 20:54:02 +0200 (Tue, 22 Jul 2008) | 1 line Remove out-of-date section on Exact/Inexact. ........ r65188 | raymond.hettinger | 2008-07-22 21:00:47 +0200 (Tue, 22 Jul 2008) | 1 line Tuples now have both count() and index(). ........ r65189 | raymond.hettinger | 2008-07-22 21:03:05 +0200 (Tue, 22 Jul 2008) | 1 line Fix credits for math.sum() ........ r65190 | raymond.hettinger | 2008-07-22 21:18:50 +0200 (Tue, 22 Jul 2008) | 1 line One more attribution. ........ r65192 | benjamin.peterson | 2008-07-23 01:44:37 +0200 (Wed, 23 Jul 2008) | 1 line remove unneeded import ........ r65194 | benjamin.peterson | 2008-07-23 15:25:06 +0200 (Wed, 23 Jul 2008) | 1 line use isinstance ........	2008-07-23 16:10:53 +00:00
Jeremy Hylton	73fd46d24e	Bug 3347: robotparser failed because it didn't convert bytes to string. ... The solution is to convert bytes to text via utf-8. I'm not entirely sure if this is safe, but it looks like robots.txt is expected to be ascii.	2008-07-18 20:59:44 +00:00
Jeremy Hylton	1afc169616	Make a new urllib package . ... It consists of code from urllib, urllib2, urlparse, and robotparser. The old modules have all been removed. The new package has five submodules: urllib.parse, urllib.request, urllib.response, urllib.error, and urllib.robotparser. The urllib.request.urlopen() function uses the url opener from urllib2. Note that the unittests have not been renamed for the beta, but they will be renamed in the future. Joint work with Senthil Kumaran.	2008-06-18 20:49:58 +00:00