com.sun.portal.providers.urlscraper
Class Fetcher

java.lang.Object
  extended bycom.sun.portal.providers.urlscraper.Fetcher

public class Fetcher
extends Object

This class fetches and stores the content from an arbitrary url


Field Summary
static String KEY_COOKIES_TO_FORWARD_ALL
          wether to Forward all Cookies as Boolean
static String KEY_COOKIES_TO_FORWARD_LIST
          if only selcted Cookies will be forwarded, the list of cookies to be forwarded as List
static String KEY_FORM_DATA
          The formData as String e.g: login=[uid]&passwd=[password] wher the characters in square bracket are properties available in the channel.
static String KEY_HTTP_AUTH
          isHttpAuth , set true only if HttpAuth
static String KEY_HTTP_AUTH_PASSWORD
          the password for Http Auth
static String KEY_HTTP_AUTH_UID
          The Uid for Http Auth
static String KEY_INPUT_ENCODING
          The Input encoding as String
static String KEY_LOGIN_FORM_DATA
          The form data to be sent to during login if provided e.g: login=[uid]&passwd=[password]
static String KEY_LOGIN_URL
          login URL as string .
static String KEY_LOGOUT_URL
          logoutUrl as string .
static String KEY_RESOURCE_BUNDLE
          As ResourceBundle
static String KEY_RULESET_ID
          The RulesetId as String
 
Constructor Summary
Fetcher(Map config)
           Constructor method
Fetcher(Map configMap, boolean ubt, String ubtAppendUrlSt)
           Constructor method
 
Method Summary
static String getContentEncoding(String contentType, byte[] bytes, String MIMEType, String profileCharset)
          Gets the charset
 StringBuffer getFilteredContent(int timeOut, javax.servlet.http.HttpServletRequest req, javax.servlet.http.HttpServletResponse res, String urlAsString)
          gets the Filtered Content within the timeOut limit else fails
 void logout(int timeOut, javax.servlet.http.HttpServletRequest req, javax.servlet.http.HttpServletResponse res)
          Invokes the logout within the timeOut limit else fails
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

KEY_RULESET_ID

public static final String KEY_RULESET_ID
The RulesetId as String

See Also:
Constant Field Values

KEY_COOKIES_TO_FORWARD_ALL

public static final String KEY_COOKIES_TO_FORWARD_ALL
wether to Forward all Cookies as Boolean

See Also:
Constant Field Values

KEY_COOKIES_TO_FORWARD_LIST

public static final String KEY_COOKIES_TO_FORWARD_LIST
if only selcted Cookies will be forwarded, the list of cookies to be forwarded as List

See Also:
Constant Field Values

KEY_RESOURCE_BUNDLE

public static final String KEY_RESOURCE_BUNDLE
As ResourceBundle

See Also:
Constant Field Values

KEY_INPUT_ENCODING

public static final String KEY_INPUT_ENCODING
The Input encoding as String

See Also:
Constant Field Values

KEY_FORM_DATA

public static final String KEY_FORM_DATA
The formData as String e.g: login=[uid]&passwd=[password] wher the characters in square bracket are properties available in the channel.

See Also:
Constant Field Values

KEY_LOGIN_URL

public static final String KEY_LOGIN_URL
login URL as string . If this is provided it will be used once to authenticate for the first time

See Also:
Constant Field Values

KEY_LOGIN_FORM_DATA

public static final String KEY_LOGIN_FORM_DATA
The form data to be sent to during login if provided e.g: login=[uid]&passwd=[password]

See Also:
Constant Field Values

KEY_HTTP_AUTH

public static final String KEY_HTTP_AUTH
isHttpAuth , set true only if HttpAuth

See Also:
Constant Field Values

KEY_HTTP_AUTH_UID

public static final String KEY_HTTP_AUTH_UID
The Uid for Http Auth

See Also:
Constant Field Values

KEY_HTTP_AUTH_PASSWORD

public static final String KEY_HTTP_AUTH_PASSWORD
the password for Http Auth

See Also:
Constant Field Values

KEY_LOGOUT_URL

public static final String KEY_LOGOUT_URL
logoutUrl as string . If this is provided it will be used logout

See Also:
Constant Field Values
Constructor Detail

Fetcher

public Fetcher(Map config)
        throws MalformedURLException

Constructor method

Parameters:
config - The Map needs lot of entries and the object type as defined.
Throws:
MalformedURLException - The Exception

Fetcher

public Fetcher(Map configMap,
               boolean ubt,
               String ubtAppendUrlSt)
        throws MalformedURLException

Constructor method

Parameters:
configMap - the MAP
ubtAppendUrlSt -
ubt - Whether to track links outside portal
Throws:
MalformedURLException
Method Detail

getContentEncoding

public static String getContentEncoding(String contentType,
                                        byte[] bytes,
                                        String MIMEType,
                                        String profileCharset)
Gets the charset

This method determines the charset based on the contentType header if it is available (only applies to http(s) urls), or from the inputEncoding property if it is non-blank, or from the meta tag in content, e.g. meta tag in html, xml or wml header if they are available (only applies to HTML, XML, WML).

Parameters:
profileCharset -
contentType - If http(s) urls, null otherwise
bytes - Bytes from the scraped content
MIMEType - MIMEType for the content
Returns:
String charset or null if charset cannot be determined

getFilteredContent

public StringBuffer getFilteredContent(int timeOut,
                                       javax.servlet.http.HttpServletRequest req,
                                       javax.servlet.http.HttpServletResponse res,
                                       String urlAsString)
gets the Filtered Content within the timeOut limit else fails

Parameters:
timeOut -
req -
res -
urlAsString -
Returns:

logout

public void logout(int timeOut,
                   javax.servlet.http.HttpServletRequest req,
                   javax.servlet.http.HttpServletResponse res)
Invokes the logout within the timeOut limit else fails

Parameters:
timeOut -
req -
res -
Returns: