master Add an extra function tokenize_with_span to WordpieceTokeniz…

* add bert tokenizer with span

* fix lint issue

* adjust endpoint

* modify length calculation

* integrate tokenize_with_span into tokenize func

* add unit test

* fix typing issue

* change output to concatenated string

* string to tuple

* unittests

* explicit return type

* type ignore

* separate into two funcs

* remove an unused var

* modify unit test

* fix format issue

* remove unused import

* remove unused import

* modify docstring

Co-authored-by: Yishan Jiao <yishan.jiao@petuum.com>

Hector avatar Hector

Push event #811 passed

  • Ran for
AMD64
no language set
Git

Sorry, we're having troubles fetching jobs. Please try again later.