MCPcopy
hub / github.com/dgraph-io/dgraph / LangBase

Function LangBase

tok/langbase.go:26–73  ·  view source on GitHub ↗

LangBase returns the BCP47 base of a language. If the confidence of the matching is better than none, we return that base. Otherwise, we return "en" (English) which is a good default.

(lang string)

Source from the content-addressed store, hash-verified

24// If the confidence of the matching is better than none, we return that base.
25// Otherwise, we return "en" (English) which is a good default.
26func LangBase(lang string) string {
27 if lang == "" {
28 return enBase // default to this
29 }
30
31 // Acquire the read lock and lookup for the lang in cache.
32 langBaseCache.RLock()
33
34 // check if we already have this
35 if s, found := langBaseCache.m[lang]; found {
36 langBaseCache.RUnlock()
37 return s
38 }
39
40 // Upgrade the lock only since the lang is not found in cache.
41 langBaseCache.RUnlock()
42 langBaseCache.Lock()
43 defer langBaseCache.Unlock()
44
45 // Recheck if the lang is added to the cache.
46 if s, found := langBaseCache.m[lang]; found {
47 return s
48 }
49
50 // Parse will return the best guess for a language tag.
51 // It will return undefined, or 'language.Und', if it gives up. That means the language
52 // tag is either new (to the standard) or simply invalid.
53 // We ignore errors from Parse because to Dgraph they aren't fatal.
54 tag, err := language.Parse(lang)
55 if err != nil {
56 glog.Errorf("While trying to parse lang %q. Error: %v", lang, err)
57
58 } else if tag != language.Und {
59 // Found a not undefined, i.e. valid language.
60 // The tag value returned will have a 'confidence' value attached.
61 // The confidence will be one of: No, Low, High, Exact.
62 // Low confidence is close to being undefined (see above) so we treat it as such.
63 // Any other confidence values are good enough for us.
64 // e.g., A lang tag like "x-klingon" should retag to "en"
65 if base, conf := tag.Base(); conf > language.No {
66 langBaseCache.m[lang] = base.String()
67 return base.String()
68 }
69 }
70 glog.Warningf("Unable to find lang %q. Reverting to English.", lang)
71 langBaseCache.m[lang] = enBase
72 return enBase
73}
74
75func init() {
76 langBaseCache.m = make(map[string]string)

Callers 5

TokensMethod · 0.85
processTokensMethod · 0.85
TokensMethod · 0.85
TestLangBaseFunction · 0.85
GetTokenizerForLangFunction · 0.85

Calls 8

RLockMethod · 0.80
RUnlockMethod · 0.80
WarningfMethod · 0.80
ParseMethod · 0.65
LockMethod · 0.45
UnlockMethod · 0.45
ErrorfMethod · 0.45
StringMethod · 0.45

Tested by 1

TestLangBaseFunction · 0.68